I had the same issue where my background would shift when I changed some of the content on top of it. I couldn't identify what the issue was but when coming back to the issue in iOS 16.4-6 I found that the shift was always present, not affected by the overlaid content anymore. Well I guess consistency is often easier to deal with!
I found another blog describing how to get a fullscreen background image and it gave me a clue that helped me solve the issue (though it doesn't actually go into this specific issue and you can actually see that earlier in the page it's affected by the issue as the moon is shifted to the right but ends up central): https://ix76y.medium.com/space-travel-app-part-1-creating-a-fullscreen-background-image-in-swiftui-beginner-tutorial-ca6f950b3b8
I was using .scaledToFill() like you but that seems to be part of the problem, if you instead use the following (as shown in the blog) it doesn't shift the image and looks like it scales ok:
.aspectRatio(contentMode: .fill)
.frame(minWidth: 0, maxWidth: .infinity)